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Four independent approaches to the formulation of a 
taxonomy of review publications and the subsequent development of 
criteria useful to their planning and evaluation are presented* One 
approach considers reviews on a continuum of criticallty ranging from 
the bibliographic review through the interpretive to the critical. 
Another considers types of reviews as information requirements 
necessary to the development of innovations or the solution of 
problems. A third alternative characterizes reviews according to the 
purpose of the producers and the consumers « A final approach shows 
how criteria may be derived from 1) the type of intellectual 
processing applied to the preparation of the review, 2) its 
characteristics (readability, currency, etc.)* and 3) its substance. 
It is shown how evaluation criteria may be developed through the 
specification of review dimensions. (Author) 
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Four independent approaches to the formulation of a taxonomy 
of review publications and the subsequent development of criteria 
useful to their planning and evaluation are presented* These 
approaches are based on the type and degrde of intellectual 
processing applied, a transfer model, the purposes of users and 
producers, and their innate dimensions. 



In recent years scholars from a number of disciplines have begun to urge the pro- 
duction of more and better review publications as a way of controlling an increased 
amount of data and information. These scholars recognize the utility of current 
efforts, particularly the- use of those that involve computer- driven information . 
retrieval systems, in supplying access to this information. They point out however, 
that the simple provision of efficient access and retrieval of pertinent documents 
does not solve the problem. Users of these systems are merely presented with a 
number of documents or citations, some containing the necessary information and 
some not. They must still carefully peruse these documents, recognize and extract 
the required information, and convert it to a form that can be absorbed and used. 
In some cases this effort taxes not only their patience but also their capability. 
An .example of the latter is the case of the practitioner who is confronted with 
advanced mathematical formulae or theories or research data. True control and 
ultimate utilization can only be brought about through the processes of purposeful 
condensation and tailoring the information to the needs of particular audiences. 
To be useful, information must be of a form and quantity that can be absorbed by 
human beings r^^" 

By themselves, retrieval systems cannot accomplish these things. They are unable 
to condense, analyze, interpret, synthesize or evaluate information, and they are 
unable to convert information into usable knowledge. Such transformations are 
commonly accomplished, however, through the process of reviewing, Reviews there- 
fore have a very basic, almost unique, role to play in the generation and in the 
ultimate utilization of knowledge. 
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One would think that In view of the importance of reviews, information scientists 
woiJ).d be diligent Ly studying their character is tics , developing a taxonomy, suggest- 
ing procedures for their preparation and evaluating and designing courses for 
training reviewers. Such is not the case. Reviews, reviewers, and reviewing have 
all been neglected by information scientists. 

In a recent literature survey covering the past ten years this author found that 
less than three dozen pertinent papers on the subjects of either reviews or re- 
viewing have been published in the English language. The list of these papers was 
circulated to a number of authorities who were able to add only three citations* 
' Compare this to the several thousand papers on indexing that have been produced! 
The dearth of scholarly studies on this important subject has prompted the author 
to share some observations and thoughts on alternative ways to distinguish types 
of reviews, to define their dimensions and to make a start toward developing \\ 
planning and evaluation criteria. These Treasures are necessary to assure quality,^ ^ 
to provide guidance to those preparing them, to establish bases for resource A 
allocation and, finally to structure the field in such a way that developed 
techniques can be taught to others. From a scholarly standpoint these measures 
can provide a start in an area rich in opportunity. 

Three possible approaches to the formulation of a taxonomy of review publications 
and the possibility of subsequent development of criteria for design and evaluation 
will be considered: (1) the degree of kind of intellectual effort necessary 
for their preparation, (2) the purposes of their producers and consumers, (3) their 
^^•ijiate dimensions, that is, their substance, characteristics and intellectual level. 
While all three approaches have their uses the last seems most promising for those 
interested in further scholarly development. 

The intent of this paper is to break ground not by providing a single unifying 
theory but by offering alternative conceptualizations. The approaches presented 
are general and are not intended to apply to any one specific subject field. 
Even though reviews prepared for administrators are a relative^' rarity in science 
and technology, reviews for administrators are included since they are fairly 
common in education. Finally in keeping with the purpose of this paper, reviews ^/ 
are. broadly defined. A review is considered to be a narrative presentation or 
reformulation of existing information on a topic derived from a variety of printed 
sources. 

Intellcc tjjal^ Proce ssing 

Discussions of reviews often center about a somewhat simplistic polarization. 
Typically, in such discussions, inonlion will be made of the need for expert or 
critical reviews in a sul)jcct lield of roji-non interest and concern will be ^ 
expressed about tiie unwi 1 1 inj^ncijs oi k':-:pert!; to prepare them. 

The point may then be made tliat whili- ihi re are plenty of reviews of > the non- 
critical type published, p.irt icuhir 1 y the bibl io>;raphic , this is not what is 
ncL'deu. i'rcdictably , interest will tlu-a turn to the **bread and butter" issue 
of h.ow experts can be rcmunerate<l or iithc^'wise encouraged to do the necessary 
v:ork. 
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The need for critical reviews is real and the remuneration issue is important. Yet 
the discussions are simplistic in that they assume there is only one ^^real" type of 
revl.ew, the critical, and presumably that the other forms are hardly worthy of the 
name. Completely overlooked are the many other types of reviews which may be pre- 
pared by persons with skills other than expert subject knowledge. 

these types differ in the kind and degree of intellectual effort necessary for their 
preparation. Clearly the lowest form is the bibliographic. A distinction, borrowed 
from the field of abstracting, might be made between the indicative bibliographic 
review, which is simply a bibliography in narrative form, and the informative. The 
latter may be considered to require greater "criticality" since significant or 
comv ^ble data myst be extracted from a variety of publications and presented. 
From tiiese two we could proceed through reviews requiring greater effort such as 
the interpretive, the state-of-the-art, the evaluative and finally reach the true 
critical review perhaps written by the outstanding expert. Critical reviews survey 
entire fields or sub-fields, analyze and evaluate developments, synthesize the 
distilled results and sug'gest the directioiT the field should take. 

The various types of reviews can thus be considered to be points that can be located 
on a continuum based on the intellectual effort that has been applied to them as in 
Figure 1. 

FIGURE 1 

REVIEHS ON A CONTINUUM OF CRITICALITY ^/ 



TYPE OF REVIEW 



EXPERT 
EVALUATIVE 
ANALYTICAL 



STATE-OF-THE-ART 



INTERPRETIVE 
POPULAR 



INFbR>lATIVE 
INDICATIVE 



MORE CRITICAL 



CRITICAL 



TOPICAL 



EXPLANATORY 



BIBLIOGRAPHIC 



LESS CRITICAL 



As one proceeds up the scale the reviews become less literature oriented and more 
topic oriented and progressively tr.on- Kopfdsi fcatcd in both the intellectual skills 
applied to them and "^the subject e\*pei ti;;c involved. • 
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This structuring has several merits. It reveals to the scholar an *4ntellec tual 
dimension" of reviews which may be capable of further development* It helps the 
designer to orient himself and, since he knows the type vjith which he is dealing, 
it provides some basis for achieving consistency. It helpy the administrator to 
distinguish the variety and degree of skills necessary for preparation of different 
publications. ^ . 

Yet the structure is far from satisfying. For one thing, we might accept the fact 
that some /eviews are "higher*' and some "lower" in intellectual level. This 
assumption has obvious validity in the case of the bibliographic versus the critical. 
Yet in the middle range one would be hard pressed to place the analytical^ inter- [ 
pretive> or synthetic on a higher or lower point of the continuum. v^' 

t 

An alternative and perhaps more useful structure would be to consider types of 
reviews on a time scale relating to users* information requirements. The scale 
would begin with the germination of an idea or the identification of a problem and 
end with public use of an adopted technique-or technology. Early reviews would deal 
with literature designed to bring the problem into focus for administrators and 
legislators and to convince them to allocate resources. These reviews may range 
from simple administrative memoranda to formal research proposals or even to popular 
reviews appearing in the mass media. If commitments are made to research the 
problem, state-of-the-art reviews are prepared. As the work proceeds into the 
research phase there may be need for the methodological and analytical and the 
synthetic reviews. As the research phase ends and adoption begins there is again 
need for reviews to interpret the research in language understandable to adminis- 
trators. Following this second type of commitment, practitioners (teachers, 
engineers, etc.) require interpretations that allow them to use the results of 
research as developed techniques or technology. Finally, popular reviews might be 
prepared that enable the general population to appreciate the advantages and dis- 
advantages of what has been accomplished so th.ey can support, reject or use it. 

Figure 2 provides an example of such a structure showing how different types of 
reviev'S are required for tasks necessary to the development of an irmovation. 
Typical audiences are also indicated. Because of space limitations only five types 
of reviews are shown, and there are, of course, many more. It should also be 
recognized that not all tasks and phases are listed, for example a development phase 
and a maintenance task might also be included. 

This developmental model with variations, "loops", and fill-ins has been found useful 
by those interested in "technolo>;y transfer" beyond the research phase. Various 
types of reviews have always bocn used within the research stage, interestingly 
enough in a less systematic way than by those who plan the transfer of techniques 
and technology. 

Both of the foregoing models, the critical and the developmental, suffer from an 
important defect. They only describe one n^view dimension. They do not reveal • 
characteristics of reviews other than tho iniellectual. Yet it is from user require- 
ments in an additional dimension, requlrinvMU s that reviews be timely, comprehensi'/e, 
authoritative, readable and so on that thoso charged with specific planning or 
preparation derive some of their most Import :int criteria. Something must be added 
to these models. 
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Perhaps Insights into other dimensions and further specification of the intellec- 
tual dimension can be arrived at by examining users', purposes in consulting or 
reading reviews. If we knew their goals perhaps we can design vehicles to reach 
there. 

Purpose of the User 

From a review of the deductions of previous authors if and the little empirical 
evidence available, it would seem that reviews perform four major functions. These 
functions are similar to those afforded by: (1) textbooks (2) alerting services,- 
or cutrent bibliographies (3) reference books and ^'finding devices" and (4) In- f 
spirational works. ) 

The textbook function accommodates those who need to achieve an understanding of an 
aspect of 5 their own field with which they have limited familiarity. Similarly it 
accommodates those who need orientation in a new or peripheral field. 

As there are a variety of textbooks available on a single topic, designed for" 
different audiences, so there are a variety of reviews that perform the same 
function. Some translate from one technical language to another. Others interpret 
material at a popular level. In these ways, reviews provide a means to achieve 
understanding regardless of the nature of the original material. On this point, 
Scott Adams noting the institutionalization of this effort in medicine, made the 
Interesting observation that in medicine, the Advances characterize the research 
level, Yearbooks the practice level, and reviews of the Scientific Monthly type 
the popular level. 1/ 

Reviews are not only used to aid in understanding new or peripheral areas or 
material on a different technical level. They may also be utilized In the same 
way a student will use a variety of textbooks, that is, to ^'see" difficult 
material from a different angle or to make it "sink-in." 

In all these ways reviews serve those who are not able or do not have the time to 
perform the intellectual processing necessary to make original papers usable for ^ 
their purposes. 

A second important function is the current bibliographic. Users, whether they be 
practitioners, researchers or teachers must maintain ^rrent awareness in their 
field if they are not to become "obsolescent." In some cases, particularly with 
researchers, the> raust> have assurance that they have not missed recent significant 
works in the course of their regular reading. When reviews are used to serve 
thesfe purposes they fulfill the same functions as alerting services or current 
bibliographies although they more ofjien go beyond the mere provision of references 
and often supply actual information sufficient to make further consultation of the 
literature unnecessary. There is some evidence that this continuing education or 
current awareness function is the most important one served by reviews. 

Indexing and abstracting services, data compilations, directories, etc. provide 
data^ information or reference. Reviews are often used for the same purposes. It 
|6 fcW the review will often provide information In a more timely way or in 

f I ^:ic.#|i;.|x t that Includes caveats, suggestions or alternatives. Despite this the 
iSdMc^^ of use is for reference. 



Finally, these publications may be used as a source oi inspiration, a vay of 
maintaining flagging interest, of stimulating work In new directions. They are 
a source of new ideas and a "creative bed" for their development. 

Actual data oh users' purposes in reading or consulting reviews is sparse. Usually 
they can be found only In bits and pieces within broadev user requirement studies. 
Recently* however, two research studies 2/i A/ ^^ve appeared that provide relatively 
comprehensive data on purposes served by two specific review publications. 

Using both a critical incident technique and a structured questionnaire to study the 
Annual Review of Information Science and Tech nology (ARIST) Carlos Cuadra arrived 
at results that tend to confirm the validity of t^ four categories* At the same 
time he provides data on relative use within eafC^ category. _5/ Some of his re- 
suits are presented in Table 1 where they are rlfearranged to show how the categories 
he discovers "fit" into the four described above. 

Percent of Readers Finding ARIST Useful 
in Serving Various Goals 



Category and Coal Percent 
Textbook 

Learning about an area not in reader^ s specialty 22 

Current bibliographic 

Keeping up with own area 33 

Keeping up with peripheral area 35 

Reading the original literature more selectively IS 

Reference 

Checking on particular project 19^ 

Checking on individuals 5 
Inspirational 

Identifying areas that require further research 14 

Allocating R£rD funds 2 

Other ^ 

Table 1 



Similarly when Itical incident technique was used responses were highest In 

the keeping current and reference areas and In responses related to learning. 

Another investigator, Linda Harris, found thit the major purposes of use of 
another review publication (the rcadorsiilp of the Rcvjcw^^ tducatlOnal Resear ch) 
were also for learning, keeping current and reference^ 6/ in this structured 
6tudy;ihi Included an additional category for "to tnoko sure I hadn^t missed ift- 
;|§|t|nt iiu^ This category dctnonstr.uos another Aspect of the current 

^biSiicgraphy approach eraphasi^ing approptJcilo ^Jplccilvity ^nd cotaptehensivencss 
"itlvlirSthan currency. 



If lt'l« difficult to find data on the purposes of the user, it is almost 
impossible to find them on the purposes of the producer. Two years ago in 
"breaking ground*' for another project, the author surveyed 20 ^directors of 
inforraation centers. The directors were asked in a very open-ended way to 
indicate their jJUrposes in producing a total of 200 review papers. Responses 
seemed to fall naturally into five general classes! one literature oriented and 
four topic oriented. They are tabulated in Table 2. 

Producers Purpose in Publish i ng Reviews 



Category Percent of responses 



Aid user to use the literature 40 

Provide analysis and evaluation 32 

Aid ^ putting research into practice 16 

Educate and stimulate 8 

Ansi^r repetitive inquiries .4 

Table 2 



These replies do not relate to a particular type of review ^s in the case of the 
periodic type in the Cuadra and Harris studies since the directors were free to 
produce any type of review they wished including the periodic. 

The results are probably skewed sind^ the centers were at that time only recently 
organized so that one would expect an over-emphasis on bibliographic reviews while 
a **data base*' was being established* Fewer topical and critical reviews would be 
expected in information centers than in other centers. 

Yet the users and producers do show considerable correspondence in their purposes. 
Aid to the user in using literature is composed of such replies as "update the 
user," "update a publication,** "provide a comprehensive compilation." It corres- 
ponds rather strongly to the reference and current bibliography approach. 
"Analysis and synthesis** relates to the textbook approach as does "aid in putting 
research into practice,** since they all involve teaching or the preliminaries 
thereto. The *'educate and stimulate*' purpose finds its counterparty in the textbook 
and inspirational types of use. 

It is certainly encouraging to find that the purposes of these publications seem 
to tdatch the purposes of their users. Thus, if a practitioner needs to have a 
summation of research translated into language he can understand (textbook 
Approach); it is essential that producers match this requir^ent. If a community f 
of researchers finds it necessary to keep current in a field peripheral to their 
own (current bibliography approach), it is well that producers fill this need. 
The categorization proposed thus provides general critoria for whether a type of 
f^SJ^ Wto he produced. It also provides some guidance for the monitor of 
iW^i^ pro^tiitfis Bud a starting point for the student of reviews # The categoriza- 
Hb^^^^ administrator a toolvfor structuring th^ requirements of the 

^iffeltplace and possible for devising strategics. . 



Yet the model is not quite satisfactory to the student of reviews. The student 
needs a way of analyzing their finer structure and of quantifying review parameters 
and user needs in order to formulate and test hypotheses. The review designer 
obviously does not need the detail required by the scholar yet he does have a need 
for finer detail than has been provided so far. 

The following section is devoted to how this can be done; how criteria for planning 
and evaluating reviews can be formed from three review dimensions: intellectual 
content, characteristics and substance. It will be shown how selection of criteria 
for design and subsequent evaluation takes place through consideration of user 
requirements in these areas and how specification and quantification of these 
criteria may be accomplished in some cases. Scholars will find implicit in the 
» discussions suggestions for further study. 

Dimensions and Criteria 

In 1959 Isabella Leitch characterized reviews as being of seven types; the peri- 
odical review, the occasional, the analytical, those that review data, those that 
review concepts^ the interpretive and the creative. T^l The periodical is dis- 
tinguished by its limited and regular time coverage, usually restricted to a year. 
The occasional review is pot distinguished by time but by the expertise or 
authoritativeness of the writer. It is intended to be selective and not compre- 
hensive. The analytical review was taken in. part to be an "... inquiry into the 
deductions that may be drawfi from an accumulation of results treated as a new 
whole.'* 8/ Data and concept reviews were obviously distinguished by the things 
th^y reviewed. By interpretive review, Leitch dtd not intend to describe one that 
translated data or theory into terms understandable to the reader but rather 
referred to the interpretation and deduction necessary on the part of the writer. 
The creative review possibly corresponds in part to the synthetic the highest 

and rarest which takes data from mor^i than one field and shows that they are 
related and what the relation is." 9/ Here she was writing of the generation of 
new hypotheses and not merely of putting a piece of research in its context. 

It is not important at this point that we agree on the definitions and formulations 
pVoposed by Leitch. At a future date we can come to an agreement on the meaning 
of Vanalysis>" "synthesis/' "Interpretation** and so on. What is important is to 
see that she was really describing reviews across three dimensions • One dimension 
is the type and degree of intellectual processing that has been brought to bear# 
The resulting content constitutes an intellectual dimension. This dimension 
• includes the condensation and transformation that has been applied to information 
or data to convert it into a form useful to a particular audience or discipline. 
Some reviews are almost lacking in this dimension in that they merely describe or 
give references to the material they cover* 

Others are rich In that they not only survey the field, identify th^i trends, and 
pinpoint the significant, but they actually create new knowledge by fonnulatlrig 
new hypotheses. * • ; 



\ 

A second dimension is apparent in her description of pG^odic reviews and 
occasional reviews where characteristics such as timeline^, periodicity and /J^ 
selectivity are mentioned. This describes a second dimension, the characteristic ^ 
dimension. Substance and other content constitutes the third dimension and in th^^ 
above case it is represented by data and concept. / 

Since 1959, there have. been other attempts to describe types of reviews. On 
occasion, new types have been uncovered. More often other names are provided for 
what is feally the same concept or new characteristics are announced. Despite 
this, what we really find in these descriptions are really combinations and re- 
formulations of and discoveries within the three dimensions. 

To give one of many examples, in 1961 Scott Adams described the Jahresberichte 
or discipline review as "... a comprehensive, descriptive record of annual 
contributions.*, not created to be critical or evaluative but to provide a 
systematized running record, year by year, of the contributions made within the 
disciplines.** 10/ Here we can see parts of all three dimensions identified and 
then combined to conceptualize an important 'type of review - and all in one simple 
statement! 

There are other indications of the validity and usefulness of these three dimensions. 
For one thing the few studies that have been published tend to group data according 
to the above categories. Menzel, 11/ for example, provides a table in which he 
groups the responses according to characteristics and intellectual content. Cuadra 
includes a table where percent of use is given for substance, 12 / Also it should 
^l)e pointed out that the dimensions coincide with the major universal facets pro- 
posed by many information scientists, namely substance, process and property. 
While additional facets are often advanced depending on the discipline, these 
three are usually proposed as the coordinates or dimensions by which any concept 
can be described or analyzed. 

As discussed above, if we consider the various ways by which reviews*" have been 
designated, we find that they are merely expressions of one or more aspects of the 
three dimensions. The following table illustrates this point. 



V 



Types of Reviews 



Dimension 



A. Intellectual content 



B. Characteristic 



C, Substance and Content 



Common Designations of Reviews 
Evaluative 
Analytical 
Interpretive 
Bibliographic 
Critical 
Creative 
Comprehensive 
Periodic 
Authoritative 
Popular 
Occasional 
Current 

Substantive designation 
Data reviews 
Review of concepts 
Methodological 
Interdisciplinary 



Table 3 



As described previouslyi these designations contain the factors that may be used 
as design and evaluation criteria. Before they can be used in this way they must 
be listed as illustrated in Table 4. . 



Speclflcatton of Dttnenslons to Form First Level Criteria 



Intellectual content 
Analysis 

ip^Pln-^polntlng significant 

Discovering shortcomings 

Discovering trends and 
patterns 

Synthesis 

Evaluation 

Interpretation 



Characteristics 
Author 1 tat Iveness 
Comprehenslvenoas 
Currency 

Periodicity 

Degree of compression 

Readability 

Ease of Use 

Table 4 



Substance 
Data 
Theory 
Methodology 

Ideas 
Citations 

Interdisciplinary 

subject 
Substantive 

designation 



The items listed within each dimension may now be considered to be first-level 
criterl^. This transformation of dimensions into criteria is a normal progression. 
Of course, to be considered as true criteria they must be progressively specified 
and .ultimately quantified* Even in the form presented, however, they become a 
shopping list useful in the planning and evaluation of the publications* Figure 3 
shows how this may be done, 

\ ■ , ■ ■ ' " ; 

Here, the users* requirements which may be rather amorphous originally are first 
defined and specified as first-^ievel criteria within all three dimensions* thus, 
a general request for information on audio-visual methods In educatipn would be 
more substantively defined. The other two dimensions would also be specified, 
Within the intellectuar cllmension users m^y n not require that the tpaterlal 

be Interpreted or evaluated. In the characteristic dimension, they may require 
periodic updating on the latest developments in the field or the potential audience 
may be composed mostly of . practitioners so that readability may be important. 

Given these requirements, an information analysis center may decide to produce a 
review for the target audience. The same criteria (in this case readability, 
currency, etc.) that were derived from an analysis of user needs may now be used 
fin the design of the review. Later evaluation would then consist of measuring the 
characteristics of the resulting publication against user criteria. 

Evaluation criteria developed in recent studies of user requirements may appear to 
differ from those presented here. Impact and some effectiveness studies seemlngj^ 
deal in an entirely different coin than that of timeliness, currency and so on. A 
tj^icifom^ in these studies would be that the user * s reading of the 

tWiM or did not tesult in an effort to obtain publications that were cited. 13/ 
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Such measures are useful and necessary but are preparatory to the development of 
the criteria listed here. To include impact studies this can be shown as in 
Figure 4 by titling the user circle "User State A** and adding another circle for 
'^User State B," This state is arrived at as a result of the user having read the 
reviawf In the case where ultimate effectiveness > e,g* resulting innovations, 
changes of direction of research projects and so on are considered, an additional 
circle would have to be drawn for ^'goal achievement. "(Figure 4) 

As in all good system work the development of criteria would be a backward 
progression from the terminal point. Thus a group of users may not attempt to ^ 
obtain publications cited In a review they have read. In order to either evaluate 
or design, this symptomatic measure must be translated back into the fundamental 
criteria/ That is, it would have to be determined that the review was not current 
or comprehensive enough or that the format and style discouraged the reader from 
thoroughly examining the publication. 

In addition to design criteria, there are a large group of "producers* criteria" 
which are outside the scope of thir. paper. *These cannot be considered to be other 
forms or alternative groupings of the design criteria. They are mostly management 
criteria derived from the objectives of the producer's organization, its resources 
and the constraints imposed on it. Additional criteria are derived from the 
literature environment, mostly the quantity and quality of existing publications 
and potential duplication or near duplication. Therefore, the decision to produce 
a review and the selection and emphasis placed on particular design criteria are 
actually derived from a variety of sources and not just user requirements. This 
Is Illustrated in Figure 5* 

Criteria from the various sources interact. To illustrate, the need for a current 
^Valuation of a particular topic may have been expressed within a subarea of 
education, and an information center may have received a number of reference 
questions on that topic. In deciding whether to publish the review the manager 
may consider that he does not have a staff member who is expert enough to prepare 
an evaluative review, nor can he find an outside specialist to do the work, 
Despite this he may judge that the topic is so significant and the amount of 
literature so large that, using less specialized personnel, he will /publish a non- 
evaluative review emphasizing currency and comprehensiveness. He notes that this 
product will be of some immediate use and that, if the bibliographic spade work is 
done, an outside specialist may be more willing to produce a review. Here we have 
an interaction between user requirements, resources and constraints resulting in 
the selection of design criteria* 

Returning to Figure 2 one can see then that in a judgment of this particular 
publication a reasonable selection of criteria would include its comprehensiveness 
and currency but not the degree of evaluation exercised, The director or manager 
can possibly be criticized for not electing lo produce an evaluative review* The 
publication itself, however, should not be criticized for being insufficiently 
evaluative since this feature was not selected as a planning criteria* 

The foregoing merely provides a' structure and a shopping list of design and first- 
i;:kv^l;f(^^ criteria that may be used in pUnning and evaluating reviews. 
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Opportunities do exist for making the list of criteria more complete, for defining 
and specifying them and, at least within the two non-substantive dimensions, for 
quantifying them. 

Benjamin S. Bloom's Taxonomy of Educational Ob^lectives, Handbook I; Cognitive 
Domainl A/ provides definitions for most of the criteria included in the intellectual 
dimension. Definitions useful to our purposes are provided for terms such as 
analysis, synthesis, interpretation, translation and others.* Also an additional 
level of specification is provided for each category* For example, ''analysis*' is 
defined and divided into three categories (analysis of elements, relationships and 
organlEational principles). This is not to say that these divisions should 
necossarily be accepted as they stand. For illustration an indication of an 
alternative type of division is included under "analysis" in Table 4. 

Th^eYe are, of course , alternative models which might be used. P. Guilford's 

familiar "structure-of-the-intellect" .model is an excellent possibility ._15/ 

This model has been used in the field of classification in connection with the 
development of "relational operators" where Farradane basis the basic mental 
processes he adopts on Guilford's findings*16/ Guilford himself incidentally took 
some issue with the Bloom taxonomy which he felt contained "an enormous amount of 
redundancy" in four of six major categories .17/ 

The characteristic dimension provides ready opportunity not only for specification 
but also for quantification. Rather sophisticated index*.; of readability that 
Involve sentence structure, content, number of syllables and word length have been 
developed and tested> 18 / 

Attempts have been made to measure currency. A useful way to Judge the currency of 
a publication is simply to consult experts to determine whether recent published 
studies are included or whether undue reliance was placed on older or even out-ofr 
date publications. This, of course, tells little about the objective age of the 
material included in the review. Some quantification can be achieved through 
tabulating the age of the citations in current bibliographies and comparing them 
With the average age of material included in the review. Similarly , "half-life" 
studies have been conducted in many fields and subfields* They provide a way of 
comparing the age of material within a publication with those in tho discipline in 
which its topic falls. 

Another opportunity for quantification of characteristic criteria may come about 
through compression ratios mentioned by Herringl9/ and Cot trell .20/ Cottrell 
proposes the use of a compression ratio (reference p^ges/state-of^the-ar t pages) 
as an evaluation criteria. Herring proposes a similar measure. With ^feviews of 
the bibliographic type, recall/precision types of tests might providQ^a measure of 
merit aJthnngh it v:ould be difficult to "fix" the universe from which the publica- 
tions were drawn. 

^ffje^tt dimension to specify is the substantive/ As suggested 11/ 

|r|ylou$ly, evaluation in this area may best be. left to the subject specialist 
yiK^ ^here ^re opportunities to develop criteria for design and evaluation* 

WHil6 little c^n be done to specify actual sabstancOj clos6 criteria and even 



specification can bKestablished for format » and statements can be made that alterna 
tive solutions to the\problem will be included as will appropriate caution?, caveats 
and constraints* A number of guides have appeared for authors of annual review 
chapters^ review Journals and government-sponsored publications that contain 
specifications for siwple^content . There are also guidelines for referees that 
contain criteria for evaluation. These guidelines which are a principal vehicle 
tot presenting authors with\aH design criteria can be improved as knowledge of 
reviews grows. 

Summary 

Four independent approaches to the formulation of a taxonomy of .review publications 
and the consequent development of criteria useful to their planning and evaluation 
have been presented. One approach considered reviews on a continuum of criticality* 
ranging from the bibliographic review through the interpretive to the crij:ical. 

.Another considered types of reviews as information requirements necessary to the 
development of innovations or the solution of problems, A third alternative 
characterized reviews according to the purpose of the producers and the consumers, 
A final approach showed how criteria may be derived from (1) the type of 
intellectual processing applied to the preparation of the review (2) its 
characteristics (readability, currency, etc*) and (3) its substance* It was 

^ shown how evaluation criteria may be developed • through the specification of review 
dimensions , 

Regardless which approach is taken, the process of .evaluation is based mainly on 
.the ^'tnatch" between user requirements and review characteristics. This is made 
possible by conceptualizing them in the same way. 

An underlying aim of this paper has been to initiate and encourage the development • 
of reviewing as a field in its own ^ight. While each of the four approaches has 
/Its special usefulness, the last mentioned 6eems to offer the most promise for the 
Academic development of the f ield^'^^rom this standpoint ' efforts should be first 
directed toward definition of the intellectual processing that takes place in 
"review preparation and requirefaents for this- processing, Th^ works of Guilford 
and Bloom offer a starting point in this regard,. * . ^ ^ 

^- • ■ ■ " " ♦ ■ ■ - 

It is hoped that interested persons will be encouraged* to begin work in this area 
so unex^'lored yet so rich in opportunity* . ^ 
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